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Method and Apparatus for synchronizing data streams contain- 
ing audio, video and/or other data 

The invention relates to a method and to an apparatus- for 
synchronizing data streams containing audio, video and/ or 
other data, wherein some of the data streams are recorded in 
a multiplex on a storage medium and other data streams are 
located out of the data stream multiplex. 

Background 

Optical storage media facilitate recording or replaying of 
audiovisual (AV) signals that include one or more video and 
audio data streams and/ or other data like subtitles and ser- 
vice information. Based on the Digital Versatile Disc (DVD) 
standard published by www.dvdforum.org pre-recorded movie 
titles are replicated for a mass market using read-only op- 
tical discs. Following introduction and proliferation of the 
red laser based DVD standard, a blue laser based media sys- 
tem of even higher capacity has recently been published un- 
der the brand name 'Blu-ray Disc 1 denoted BD. 

For nowadays DVDs the content providers create videos for 
many different target countries. This encompasses many dif- 
ferent languages and sub-titles. Additional to this lan- 
guage specific versions of AV material there exists the ap- 
plication of multi -angle and multi-story for optical discs. 
In BD applications this possibilities will be enlarged by 
introducing different video formats as SDTV and HDTV videos. 
For optical recording and pre-recording all the different 
elementary streams listed above (video-angle, audio-track, 
subtitle etc.) have to be packetized into a single multiplex 
transport stream. During playback the decoder is reading 
this multiplex and decodes those packets belonging to the 
streams selected. 
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Invention 

The invention is based on the recognition of the following 
fact. Having an AV multiplex carrying all the available com- 
s ponents it is not possible to provide additional material, 
like another sound track or another web-page for a later 
time- I.e. when a content provider is using an authoring 
tool for the generation of a BD movie all elementary streams 
have to be known before starting the generation. It is not 
10 possible to add another elementary stream after the AV mul- 
tiplex production. On the other hand , -if a further data 
stream shall be added later on, this requires a synchroniza- 
tion with the already existing data streams. 

15 Therefore , a problem to be solved by the invention is to al- 
low synchronizing of data streams recorded in a multiplex on 
a storage medium with further data streams located out of 
this data stream multiplex. 

20 This problem is solved by the method disclosed in claim 1 
and the corresponding apparatus in claim 8 . 

In principle, the inventive method allows synchronizing data 
streams containing video, audio and/ or other data, wherein 
25 some of the data streams are pre-recorded in a multiplex on 
a storage medium, wherein a navigation file comprises de- 
scriptors pointing to parts of said data streams, wherein 
said descriptors define the arrangement in time for said 
data streams by means of data sub stream paths. 

30 

Advantageous additional embodiments of the invention are 
disclosed in the respective dependent claims. 



35 Drawings 

Exemplary embodiments of the invention are described with 
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pie of a play list encompassing different video , audio and 
subtitle stream paths. 

Exemplary embodiments 

Figure 1 shows an example of a play list List_of_PlayItems 
encompassing different video, audio and subtitle stream 
paths with Playltems or SubPlayltems, which have to be de- 
coded for playing back of a stream path. A first stream path 
of Playltems comprises an AV multiplex stream. Furthermore, 
several SubPlayltems describe audio, video and subtitle 
stream paths as depicted. 

As shown, multiple stream paths may exist in parallel on the 
global time axis of the PlayList. A stream path may be de- 
fined leaky by leaving parts of the time axis empty. 

Any stream described by a Playltem or a SubPlayltem may be 
an elementary stream or again a multiplex of streams. 

The inventive synchronization of elementary streams being 
located out of the main AV multiplex on a. optical disc is 
described in the following . 

Synchronization is thought of for two different cases. The 
first case is the synchronization of components concerning 
their relative relation in time. It is- determining to which 
time the separate components start and end. The second case 
is the synchronization of components concerning their 
switching. It is determining points in time and in the bi- 
nary stream were the decoding of one component can be sub- 
stituted by decoding another component. The switching of a 
video components for example is useful for multi -angle ap- 
plications . 
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The preferable medium for such an application will be beyond 
DVD capabilities e.g. a blu-ray disc to provide the transfer 
rate needed for out of multiplex bit-stream reading. 

The out of multiplex semantic provides the ability to play- 
back blu-ray disc AV streams combining different elementary 
streams inclusive a combination of streams coming from disc 
and via internet or another storage medium. E.g. there 
could be a playback of a movie where the video stream is 
read from a blu-ray disc the audio stream is read from a 
hard disc and the .sub-title is read from- an internet ser- 
vice. 

The description for out of multiplex AV material consists of 
different files including navigation files and stream files . 
• The navigation files provide all entry points for navigation 
within the stream files as there are points in the time 
axis, angle of view, sub- titles and audio channels etc. In 
a higher layer of the navigation files there exists a list 
of play items explaining all stream parts belonging to a 
playback of the AV material. 

In the following a generic syntax for higher level naviga- 
tion files is described. In the tables 1 till 4 only those 
elements are outlined which are mandatory for this inven- 
tion, i.e. additional elements not described here as well as 
different orders may be possible. 

As defined within Table 1 a playback element on disc which 
might be any part of a movie (e.g. a chapter) is described 
by means of a Li st_of_PlayI terns . It consist of length indi- 
cating the complete length of the structure in byte, play- 

.^9W^.iJliAis»tiik3„t3fs number .of ^laylteoilL element s^axid _sjub_- 

Playltems indicating the number of SubPlayltems () within the 
List_ofJ?layItems structure. While at least one Playltem is 
mandatory for a List_of_PlayItems the SubPlayltems are op- 
tional. The time axis of SubPlayltem(s) is referring to the 
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time axis of the Playltem (s) . 



Table 1 : List_of_Playltems - Syntax 



Syntax 


No. of bits 


Mnemonic 


L.isi_OT riayiiernsi) \ 






lengin 


16 


uimsbf 


nlavl terns 

\J\Gky Uwl i icy 


1 Q 


UimSOr 








subPlayltems 


16 


uimsbf 








for f n =1 : n< playltems; n++) f 






PlayltemO { 












> 






for (n =?1 ; n < subPlayltems; n++) { 






SubPlavltemQ f 






• • • 


















> 







As defined within Table 2 a Playltem structure consists of 
length indicating the complete length of the structure in 
byte, StreamFile describing a link to the elementary stream 
file, Start_time describing the presentation start time of 
the Playltem within the StreamFile and End_time describing 
the presentation end time of the Playltem within the Stream- 
File. 

Additionally it consists of the element "reserved" to pre- 
serve byte alignment. Seamless jpresentation_f lag indicating 
is if this Playltem provides seamless transitions between com- 
ponents. The Seamless_presentation_ flag has the same mean- 
ing as within the SubPlayltem (Table 3) and is explained in 
more details there. 
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Table 2: Playltem - Syntax 



Svntax 


No. of bits 


Mnemonic 


Plavltem 0 / 






length 


16 


uimsbf 


StreamFile 




• • • 








Start time 


32 


uimsbf 


End time , , 


32 


uimsbf 








reserved 


11 


bslbf 


Seamless presentation flag 


1 


bslbf 


} 







As defined within Table 3 a SubPlayltem structure is very 
similar to the structure of a Playltem. It consists of 
5 length indicating the complete length of the structure in 
byte, StreamPile describing a link to the elementary stream 
file, Start_time describing the presentation start time of 
the Playltem within the StreamPile and End_time describing 
the presentation end time of the Playltem within the Stream- 
10 Pile. 



Additionally it consists of the element "reserved" to pre- 
serve byte alignment, Seamless_jpresentationJE lag indicating 
if this Playltem provides seamless transitions between corn- 
is ponents, Streamjpath_end indicating the end of a sub stream 
path and SubStream_type an one 8 -bit field indicating the 
type of sub stream path given for the SubPlayltem. 
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Tables: SubPlayltem - Syntax 



Syntax 


No. of bits 


Mnemonic 


SubPlayltem(){ 






length 


16 


uimsbf 


StreamFile 


... 




• ■ ■ 






Start time... 


32 


uimsbf 


End time... 


32 


uimsbf 








reserved 


6 


bslbf 


Seamless presentation flag 


1 


bslbf 


Stream path_end 


1 


bslbf 


SubStream type 


B 


bBlbf 


• a • 






> 







As defined within Table 4 the SubStream^typ® defines the 
type of SubPlayltem, This can be an auxiliary audio stream 
path for audio dubbing, a video stream path, an audio stream 
5 path, a subtitle stream path or a graphics stream path. The 
value 6 till 255 can be used for future stream path formats. 



Table 4: SubStream_type 



SubStream type 


Meaning 


0 


reserved for future use 


1 


Auxiliary audio stream oath 


2 


Video stream path 


3 


Audio stream path 


4 


Subtitle stream path 


5 


Graphics stream path 


6-255 


reserved for future use 



10 An alternative definition of the 3ubStream_type is possible 
as shown in Table 5. The advantage of divining an auxiliary 
transport stream path instead of different elementary stream 
paths is, that any elementary stream can again be embedded 
within another multiplex stream, i.e. several different sub- 
is titles may be multiplexed together within a single file. 
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Table 5: SubStream_type 



SubStream type 


Meaning 


0 


reserved for future use 


1 


Auxiliary audio stream path 


2 


Auxiliary transport stream path 


3-265 


reserved for future use 



Seamless_presentation_f lag supports the case of synchroniza- 
tion concerning seamless switching of components during 
s playback. 

The Seamless_presentation_f lag is located within the Play- 
It em/ SubPl ay it em and indicates if the transport stream ref- 
erenced to by Playl tern/ SubPlayl tern is supporting seamless 
10 transitions between its elementary streams. If Seam- 

less_presentation_f lag is set "true", all elementary streams 
within the transport stream obey the seamless transition re- 
strictions . 



is The seamless transition restrictions are providing Splice 
Points within the elementary stream. Splice Points are 
points at which the decoding of one elementary stream can be 
stopped and the decoding of another can be started without 
having any noticeable effect during playback, presuming that 

20 both elementary streams have been encoded by obeying the 

seamless transition restrictions. The generation of Splice 
Points is reached during encoding by defining a mandatory 
GOP raster, e.g. a fixed GOP length, and by limiting the re- 
quired buffer size, to avoid buff er overf lows when switching 

25 different elementary streams during decoding. 

A typical application for seamless transitions are multi- 
angle videos. A multi-angle video provides different camera 

~ angles for* "a video" e.g. one "camera " witfiXrT tfie""face - car , * axv^ 

30 other showing the box, one showing the finish line etc. 
When the multi-angle video is played back the user can 
switch seamless between this different camera tracks. 
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Streamjpath^end and SubStream_type support the case of syn- 
chronization concerning the relative relation of components 
in time * 

5 

Stream_path_end: This flag indicates the end of a stream 
path. A stream path is an ordered set of SubPlayltems. It 
defines the number and order of all SubPlayltems belonging 
to the stream path. The first stream path starts with the 

io first SubPlayltem and ends with the first SubPlayltem having 
the Stream_path_end flag set "true". The second stream path 
starts with the first SubPlayltem following the previous 
stream path and ends with the first SubPlayltem having the 
Streamjpath_end flag set "true", and so on. The last stream 

is path ends with the last SubPlayltem. Setting the 

Stream_path_end flag "true" for the last SubPlayltem of the 
PlayList is optional. All SubPlayltems of a stream path 
shall have the same SubPlayItem_type. 

20 Playing back a stream path will decode the consecutive set 
of SubPlayltems belonging to the stream path including ei- 
ther the first SubPlayltem having the Streamjpath_end flag 
set "true 7 ' or the last SubPlayltem of the PlayList. 

25 The Transport Stream (TS) is organized according to the 

MPEG-2 Systems atandard as specified in ISO/IEC 13818 -X, Ge- 
neric coding of moving pictures and associated audio infor- 
mation. However, in order to guarantee a conflict free play- 
back during decoding and at the same time a late and inde- 

30 pendent addition of content, e.g. the adding of auxiliary 
subtitles after the main authoring has already been final- 
ized, the following rules of restrictions for the transport 
streams are necessary; 



35 The TS of Playltems must contain a Program Association Table 
(PAT) and Program Map Table (PMT) . They provide the PID ta- 
ble and language information for every TS component (elemen- 
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tary scream) within the multiplex. 

The TS of SubPlayltems contains auxiliary streams. The pre- 
ferred format of auxiliary stream© again is a TS . 
Within the TS of a SubPlayltem no PAT is allowed. This pre- 
5 vents conflicts with the TS of the Play I tern. 

The TS of a SubPlayltem must contain a PMT whose PID is not 
used within the TS of PI ayl terns. 

All PID associated to elementary streams within all TS for 
Playltems as well as SubPlayltems must be different. 

10 

The advantage of the TS rules listed above is, that a single 
PID filter is sufficient for decoding all elementary 
streams. When violating the TS rules a PID re-stamping be- 
fore buffering the input data or the separating of different 
is buffers becomes necessary. 

The invention has several advantages: 

It provides more flexibility for the authoring of pre- 
recorded blue-ray discs. E,g, after the finalization of the 

20 main AV multiplex, additional languages can easily be added 
to the disc at a later time. This is because an auxiliary AV 
multiplexes containing e.g. audio and sub-title information 
in other languages can be prepared independently and added 
to the disc image without changing the basic AV multiplex 

25 (no recoding, re -multiplexing required) . 

There are no limitation for the maximum number of audio, 
subtitle , video angles, AV formats etc. caused by the band- 
width given for the multiplex. The limitation is simply de- 
30 fined by the volume capacity. 

It allows optional binding of external sources to disc con- 
• tent e.g. a special sound track from hard disc or internet 
server, resulting in added value by binding new streams to 
35 pre-recorded discs. 

Furthermore, rules are provided for simplified decoding of 
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elementary screams coming from different transport streams 
and for seamless stream switching (e.g. for multi-angle vid- 
eos) across different transport streams. 
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1. Method for synchronizing data streams containing video, 
audio and/or other data, wherein some of the data streams 
are pre-recorded in a basic AV data stream multiplex on a 
storage medium, wherein a navigation file 
(List_of_JPlayI terns) comprises descriptors (Playltem, 
SubPlayltem) pointing to parts of said data streams, 
wherein said descriptors define the arrangement in time 
for said data streams by means of data sub stream paths . 

2 . Method according to claim 1, wherein some of the data 
streams are located out of the basic AV data stream mul- 
tiplex on the storage medium. 

3. Method according to claim 1 or 2, wherein said descrip- 
tors define the synchronization of components concerning 
their relative relation in time. 

4. Method according to claim 3, wherein the start and end 
time of the separate components is defined. 

5. Method according to claim 1 or 2, wherein said descrip- 
tors define the synchronization of components concerning 
their switching. 

6. Method according to claim 5, wherein points in time and 
in the binary stream were the decoding of one component 
can be substituted by decoding another component are de T 
fined. 

7. Method according to any of claimB 1 to 5, wherein the 
format of the Playltems and SubPlayitems data streams is 
an MPEG transport ~st^^ stream of a 
SubPlayltem does not contain a Program Association Table 
but a Program Map Table whose PID is not used within the 
data streams of Playltems. 



11. OKT. 2002 14:04 D, LJkEATENT DEPARTMENT GERMANY : ■ NR. 3845 8. 22 

PD020099-Ri-:i^B02 



13 



8 . Apparatus for performing a method according to any of 
claims l to 6. 



11 f\\. x 1 i • AC 
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Abstract 



Several data streams contain video, audio and/ or other data. 
Some of the data streams are pre-recorded in a multiplex on 
5 a storage medium while other data streams are located out of 
the data stream multiplex on the storage medium. The data 
streams are synchronized using a navigation file 
<List_of_PlayItems) , which comprises descriptors (Play- 
Items, SubPlayltems) pointing to parts of said data streams, 
10 wherein said descriptors define the arrangement in time for 
said data streams by means of data sub stream paths - 
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